{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.3.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text generation with LSTM\n",
    "\n",
    "This notebook contains the code samples found in Chapter 8, Section 1 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "[...]\n",
    "\n",
    "## Implementing character-level LSTM text generation\n",
    "\n",
    "\n",
    "Let's put these ideas in practice in a Keras implementation. The first thing we need is a lot of text data that we can use to learn a \n",
    "language model. You could use any sufficiently large text file or set of text files -- Wikipedia, the Lord of the Rings, etc. In this \n",
    "example we will use some of the writings of Nietzsche, the late-19th century German philosopher (translated to English). The language model \n",
    "we will learn will thus be specifically a model of Nietzsche's writing style and topics of choice, rather than a more generic model of the \n",
    "English language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data\n",
    "\n",
    "Let's start by downloading the corpus and converting it to lowercase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpus length: 600893\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "import numpy as np\n",
    "\n",
    "# path = keras.utils.get_file(\n",
    "#     'nietzsche.txt',\n",
    "#     origin='https://s3.amazonaws.com/text-datasets/nietzsche.txt)'\n",
    "path=\"./nietzsche.txt\"\n",
    "text = open(path).read().lower()\n",
    "print('Corpus length:', len(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Next, we will extract partially-overlapping sequences of length `maxlen`, one-hot encode them and pack them in a 3D Numpy array `x` of \n",
    "shape `(sequences, maxlen, unique_characters)`. Simultaneously, we prepare a array `y` containing the corresponding targets: the one-hot \n",
    "encoded characters that come right after each extracted sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sequences: 200278\n",
      "Unique characters: 57\n",
      "Vectorization...\n"
     ]
    }
   ],
   "source": [
    "# Length of extracted character sequences\n",
    "maxlen = 60\n",
    "\n",
    "# We sample a new sequence every `step` characters\n",
    "step = 3\n",
    "\n",
    "# This holds our extracted sequences\n",
    "sentences = []\n",
    "\n",
    "# This holds the targets (the follow-up characters)\n",
    "next_chars = []\n",
    "\n",
    "for i in range(0, len(text) - maxlen, step):\n",
    "    sentences.append(text[i: i + maxlen])\n",
    "    next_chars.append(text[i + maxlen])\n",
    "print('Number of sequences:', len(sentences))\n",
    "\n",
    "# List of unique characters in the corpus\n",
    "chars = sorted(list(set(text)))#预料中唯一字符组成的列表\n",
    "print('Unique characters:', len(chars))\n",
    "# Dictionary mapping unique characters to their index in `chars`\n",
    "char_indices = dict((char, chars.index(char)) for char in chars)\n",
    "\n",
    "# Next, one-hot encode the characters into binary arrays.\n",
    "print('Vectorization...')\n",
    "x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)\n",
    "y = np.zeros((len(sentences), len(chars)), dtype=np.bool)\n",
    "for i, sentence in enumerate(sentences):\n",
    "    for t, char in enumerate(sentence):\n",
    "        x[i, t, char_indices[char]] = 1\n",
    "    y[i, char_indices[next_chars[i]]] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the network\n",
    "\n",
    "Our network is a single `LSTM` layer followed by a `Dense` classifier and softmax over all possible characters. But let us note that \n",
    "recurrent neural networks are not the only way to do sequence data generation; 1D convnets also have proven extremely successful at it in \n",
    "recent times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import layers\n",
    "\n",
    "# 下面是构建模型\n",
    "model = keras.models.Sequential()\n",
    "model.add(layers.LSTM(128, input_shape=(maxlen, len(chars))))\n",
    "model.add(layers.Dense(len(chars), activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our targets are one-hot encoded, we will use `categorical_crossentropy` as the loss to train the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.RMSprop(lr=0.01)\n",
    "model.compile(loss='categorical_crossentropy', optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the language model and sampling from it\n",
    "\n",
    "\n",
    "Given a trained model and a seed text snippet, we generate new text by repeatedly:\n",
    "\n",
    "* 1) Drawing from the model a probability distribution over the next character given the text available so far\n",
    "* 2) Reweighting the distribution to a certain \"temperature\"\n",
    "* 3) Sampling the next character at random according to the reweighted distribution\n",
    "* 4) Adding the new character at the end of the available text\n",
    "\n",
    "This is the code we use to reweight the original probability distribution coming out of the model, \n",
    "and draw a character index from it (the \"sampling function\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample(preds, temperature=1.0):\n",
    "    preds = np.asarray(preds).astype('float64')\n",
    "    preds = np.log(preds) / temperature\n",
    "    exp_preds = np.exp(preds)\n",
    "    preds = exp_preds / np.sum(exp_preds)\n",
    "    probas = np.random.multinomial(1, preds, 1)\n",
    "    return np.argmax(probas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Finally, this is the loop where we repeatedly train and generated text. We start generating text using a range of different temperatures \n",
    "after every epoch. This allows us to see how the generated text evolves as the model starts converging, as well as the impact of \n",
    "temperature in the sampling strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1\n",
      "WARNING:tensorflow:From /home/appleyuchi/anaconda3/envs/Python3.6/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 168s 837us/step - loss: 1.9809\n",
      "--- Generating with seed: \"s\n",
      "this element as a gratuity to whatever he does as a duty; \"\n",
      "------ temperature: 0.2\n",
      "s\n",
      "this element as a gratuity to whatever he does as a duty; the self-has the believes the self-in the self-in the generation to heart and the dees to the are to have the self-conterience of the self-concention the more and the self-refure the self-concertance of the reselt to the compathing the self-concertion to heart and the well to seem the eartion of the self-concence to the self-in the self-in the self-some and the self-concertance to heart to the sel\n",
      "------ temperature: 0.5\n",
      "n the self-some and the self-concertance to heart to the self supicial into the henger and is a allofine things the self-concertion possible, the free of the repertion and them a the opinion and this to something a general to same in the seld to the to lather say who hellection, to the self-some this inturtions and dees of the part to feeling to man in a desting in the to the certain the present in the well in the refire and comences, to the intermand in t\n",
      "------ temperature: 1.0\n",
      "n the well in the refire and comences, to the intermand in the miration, our of all christive,\n",
      "to own the vincemes itsemps has been with feele toslomis of the sainteal as she nive bil be deepons theye realling to antitionalicess), the serve in men even a componatered are most armofuinal to keanry he nevermin,, this ise\"ciely doein-mishence arongly is a decigation and or unclongutionous, the greate, their was\" tas blon it ane foreavincling\n",
      "the inselfiance o\n",
      "------ temperature: 1.2\n",
      ", their was\" tas blon it ane foreavincling\n",
      "the inselfiance of obe wi n nocars to men things theses\n",
      "to netepequalligely, the skeplen to the created of to\n",
      "bemirmplead..\" he spire most which perhapsseslyleshiby enterrutence, which sictionatior is libedinglyito-myshernt to k ove, the exteption itsel\n",
      "asines, gespeth to surve tood, toluy,\n",
      "\"nreepsersk\n",
      "hithater y, be praked of the largont has centuinabline thisdly\n",
      "limoosap eny\n",
      "affect to subzaupe. that \"themxal,\"--\n",
      "epoch 2\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 185s 924us/step - loss: 1.6188\n",
      "--- Generating with seed: \"eschylus would have half-killed himself with laughter\n",
      "or irr\"\n",
      "------ temperature: 0.2\n",
      "eschylus would have half-killed himself with laughter\n",
      "or irreciation of the soul and the soul one and the to the subtines of the soul to the interression of the soul the strength of the soul the soul and the well the love of the soul and the entlier the strength of the more one and the soul and the same the stind to the order to the pressions of the precisely the soul of the suffering the subther and the world be interression of the soul and the strength o\n",
      "------ temperature: 0.5\n",
      "and the world be interression of the soul and the strength of themselves, seecity for the taste of the historicic in the more the could are soul the tradic and or as the precisely the strunger of the most substine in the one more the good or the are because of the manifest, the pering it order in the supposing in the even to be the god of many to the most so promse of the most perhaps and the restive in the world of the some supposition of the and the god \n",
      "------ temperature: 1.0\n",
      "ive in the world of the some supposition of the and the god which one, times that the con vingure to its incelted something in viewives point blous--and perhaps, and the\n",
      "a moral, distures of sydventling lines as allome one to knowile, the intelleging that casonatic, in certainy wething to men perhaps, he benore habous pronde\"--and taustor; the efficus, its great the tradly merically must men: there old one heaver libers \"orded in disdaction the have for th\n",
      "------ temperature: 1.2\n",
      "e old one heaver libers \"orded in disdaction the have for the quite think just-w--gnvavity of man were eveny-nature, wo in which to centuphtiure certure of certain, propolyquity---  their restous as farsay bating in the rea resty to lowgt and in someeghing phritsing--which estipls and to fitss--naourest of do doubed life, has\n",
      "immininafy, in xintty--whichims; their tymine .think are..\n",
      "\n",
      "radipting soul or bad of certaid pantly man sensupoles only-as re, whre \n",
      "epoch 3\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 180s 900us/step - loss: 1.5268\n",
      "--- Generating with seed: \" discharge its strength--life\n",
      "itself is will to power; self-\"\n",
      "------ temperature: 0.2\n",
      " discharge its strength--life\n",
      "itself is will to power; self-and so the sense of the stronger and so the spirit of the stronger of the stronger the sense of the stronger the spirits and the same such a more to present the spirits of the contemption of the many pain to the destines of the stronger and the spirit of the destruces of the stronger of the many says and the sense of the sense of the sense of the strengthy and the sense of the spirit and the sense\n",
      "------ temperature: 0.5\n",
      "e of the strengthy and the sense of the spirit and the sense arguther to whatever as the many parting the exertion, as the more for deeper in the contrary to prevailed in the contemption of the prould in the sempations, the most sayed in the experience present they called and philosopher has\n",
      "accompantous ever philosopher, a men individuated of the misunderstand so the most belongs and the senses to should spirit to a spirits before their many bad and so al\n",
      "------ temperature: 1.0\n",
      "o should spirit to a spirits before their many bad and so all the own men. itter a\n",
      "men? rulder--about accutrity of well list the dight of\n",
      "the vertengeed, as believes opinions; should was pwilled assumes ot feelings of deinw other through\n",
      "exercies subjedoctant\n",
      "of a\n",
      "doct nagresly calblenistry as a recoproed--that pus must \"passion to severing vasine\n",
      "ear pased\n",
      "tim day bad to heart these dows\n",
      "how vain,  finters of the echiest our extent of to under tapapistict\n",
      "------ temperature: 1.2\n",
      ",  finters of the echiest our extent of to under tapapisticts and cradpans\n",
      "and freacfly. civery sentiguerby,\n",
      "too he which. a cheicus, theik are\n",
      "seguanffells--things of\n",
      "our bad soever; in asywinestable more himpulote acchersly,\" inxidenty and\n",
      "lomisafer. no oppires\n",
      "and very europe.\n",
      "the\n",
      "towordating to lestains the liarn \"neeshy oftcom\"-pointion inhespee called probacre-and\n",
      "dindenest thele acamentarmencey of many\n",
      "recondine prouded, (for into mays donaingous hy\n",
      "epoch 4\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 176s 881us/step - loss: 1.4789\n",
      "--- Generating with seed: \" foreign, just as seems good to one, or \"as the heart desire\"\n",
      "------ temperature: 0.2\n",
      " foreign, just as seems good to one, or \"as the heart desire the state of the strong and the strong and the sense of the strong the standment the states, and the standment the standment the standment the state of the strong and the states and the strong and the strong and the stood and the standment in the strong and such a sense and the state of the strong the sense and such a still a still a still the compured the standment the seem and the strong the st\n",
      "------ temperature: 0.5\n",
      "ll the compured the standment the seem and the strong the strong that is a promate for even and everything in present and all strong, which is an actures, and that the philosophy could as the us and short, the schoo fact exaction and in the has a precisely and the will all an assumes, the fact does of the preferent and still man and himself for a science to the other seems that the more still for the contrary for the corrite of the exploness and dischas an\n",
      "------ temperature: 1.0\n",
      "the contrary for the corrite of the exploness and dischas and taded for ascevery that to fullow impatial the physiolifally even-plause is the inolings, like and or foon a matters to chount erthing the plate, in the cases the rest of a facts, it is no concernate conscioussed, standistory one's remans must hatt smart therewere, which wis\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "formior, \"which ammy to see  has love among the \"stood for fraiw exocility of truth,\" than original\n",
      "his will, to findacly,\n",
      "------ temperature: 1.2\n",
      "iw exocility of truth,\" than original\n",
      "his will, to findacly, is things.\n",
      "\n",
      "2he enor auntitu, have man: hoosm ado them ofcent that shand\n",
      "in my failh, foncever pitually has, sist at\n",
      "the koubt at even that we that it -, they stort\", here adtumentmest,--the ideals, perhaps no preach of\n",
      "allays, fyreed somerhing sho same love\n",
      "withoun the\n",
      "rear\n",
      "infect.\n",
      "\n",
      "rather, cevery cate, unmusts if in requiren qualtated for great thus understandiman wisk pattiness cottract in pre\n",
      "epoch 5\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 176s 880us/step - loss: 1.4485\n",
      "--- Generating with seed: \"n the good conscience of his tolerance, in the\n",
      "unsuspecting,\"\n",
      "------ temperature: 0.2\n",
      "n the good conscience of his tolerance, in the\n",
      "unsuspecting, and that is all the sense of the man is in the sense of the self some in the most self self sole the self sole the sense of the most storition of the stronger and the self-destruction of the state of the same the state of the stronger and the man is a sole that is as it is the stronger and solitude of the same and still the self so man is the self solitude of the sense of the spirit and self self\n",
      "------ temperature: 0.5\n",
      "s the self solitude of the sense of the spirit and self self extent him with the state of the seem of the masters and all the high so man is a consequences and present which so man\" which is sense of the stronger and expection of the condition of all things, which has have an are that unterminate and sense of the individualicy and that is the sympathy and complete of can from with delication of the compleme of his feal of which man will the stronger that a\n",
      "------ temperature: 1.0\n",
      "e compleme of his feal of which man will the stronger that a still aurage, and that truth, and man\n",
      "a still upon very with philosophy\n",
      "sense.\n",
      "\n",
      "rorch.\" the most mirbes, now, it is so even consequences; every bith, the couss of sulsict truthce with cause in\n",
      "some fingliars\n",
      "and\n",
      "ascecbated wort nature, will sarde. ethics, wo, oppose new shakes,\n",
      "and so man may never have the sud, we good which the german\n",
      "defire at could nee? thre in singles. no much\n",
      "co me the yest\n",
      "------ temperature: 1.2\n",
      "defire at could nee? thre in singles. no much\n",
      "co me the yesteriate ide every cormeries, is that\n",
      "the swacious necessary, aw diffirent\n",
      "over)_ but natural propeal afceded are can more news-dacgoverness.\"\n",
      "\n",
      "122y\n",
      "a; the alrea peter apower rainey it, throre\" beyond micror varicis, orian fack affection of\n",
      ", from nothifor rarat. conitary instinctsful whil.\n",
      "          quit \"phenosor\" con loikes-oully amlcionsity\n",
      "was\n",
      "the not opitives of main\", moral narity,\n",
      "eughle wro\n",
      "epoch 6\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 181s 905us/step - loss: 1.4247\n",
      "--- Generating with seed: \"the countries where european influence\n",
      "prevails in europe pe\"\n",
      "------ temperature: 0.2\n",
      "the countries where european influence\n",
      "prevails in europe personal and every soul and the sense of the most such all the self such a soul and all the sense and the problem of the conception of the sense of the problem of the soul and the problem of the way, and the most such a person to all the self-interests of the present to the sense and consequence and soul and the problem of the most also the world and such a soul and the most pression of the same tha\n",
      "------ temperature: 0.5\n",
      " world and such a soul and the most pression of the same that has in\n",
      "the religious to have new superiouries and faith in all life of a meaning eternally and to the most conscience our well the heart and cause it is an agreated from god proper to science and interression of the same general were one would again: the concerning to the most desires of the forevent and some fact of the all of the has has has too must are by respect, and more into else himself \n",
      "------ temperature: 1.0\n",
      "has has too must are by respect, and more into else himself and ihnevise.\n",
      "the should not would once seurs regard in\n",
      "the crutlient,\n",
      "man\n",
      "instices e sociale means to souls, and his\n",
      "exist\n",
      "attained,\n",
      "the how sciencely, is our\n",
      "endsing arg other perileby such ael, and megrance\n",
      "sours of man\n",
      "always superientlesl\n",
      "roticnare? in thereby tence of consinery has good\n",
      "our being who saindless if templly beduaring its this fasurous of home nearousae in its objecthatily,\"\n",
      "alw\n",
      "------ temperature: 1.2\n",
      "ts this fasurous of home nearousae in its objecthatily,\"\n",
      "always kindant,\n",
      "so the renating, a inganie absuld, and i drey crage\n",
      "in us eurofos. and\n",
      "appear\n",
      "ad by his know as the sin on our really now, attained, his har,, theory narrow\n",
      "howaily\n",
      "bug-inne, is with\n",
      "the\n",
      "spirit: in skepted by a, power out of my \"howeiontily groutable)\n",
      "hutagah be naces. theyemoryday s strength \n",
      "evolutive\n",
      "and justace exolque is .    ateduf?\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "1se in the right and where they aller ved \n",
      "epoch 7\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 174s 870us/step - loss: 1.4078\n",
      "--- Generating with seed: \"ample to sexual relations,\n",
      "which, as the privilege and gatew\"\n",
      "------ temperature: 0.2\n",
      "ample to sexual relations,\n",
      "which, as the privilege and gatew the world and the strength and the antiquity of the consciences of the consciences of the same the conscience of the same the consciences of the most destrice and the strength the senses of the privile of the same the animal and the most destiny of the most destroy the conscience of the most soul and the superstition of the present the most conscience of the conscience, and the conscience of the \n",
      "------ temperature: 0.5\n",
      "ost conscience of the conscience, and the conscience of the superious,\n",
      "stronger has not one ruse and forms with ourselves, and\n",
      "moral characteristic and the strength the most deception of the most present the intellections\n",
      "and the feelings of an anchimates\n",
      "really such the christian one are even pleasure the sexiousness of deminance and love the world the most strength, threatener themselves the fundamentally any death the most means of form of the consequen\n",
      "------ temperature: 1.0\n",
      "damentally any death the most means of form of the consequent other,\n",
      "a\n",
      "condition of destwity, inspired mnd\n",
      "another our first almost imblungebly for that all the much, loven\" (without sting the sslight,\n",
      "nofferoral which he \n",
      "stand--the priortion before\n",
      "that which me more other which we aunto any in liberation of fagulardder aways of\n",
      "the aniating\n",
      "asdes honongal has sentiment that we heavwerneless: but the i pecimilatores, she give strenges for the, the right \n",
      "------ temperature: 1.2\n",
      "t the i pecimilatores, she give strenges for the, the right in minds\". nevertheled--nige themstive with it\"-ecunt, with a\n",
      "pui lafiines the ge strongerans foolume of the\n",
      "manys becausiver, \"if--no manuilgious best should-blooding, oucherve\n",
      "dorsist hurg; and party of dare words hypo\n",
      "right meen,\n",
      "much\n",
      "virtues scunds,\"\n",
      "and\n",
      "ever roming tonguth here im.nval sudden\n",
      "\n",
      "i what\n",
      "offet wicch themself\" peems immoncapspromething, cained\n",
      "lame stilld of germans generatoric, h\n",
      "epoch 8\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 194s 971us/step - loss: 1.3933\n",
      "--- Generating with seed: \"itical revolutionists, but that is because we\n",
      "are in the hab\"\n",
      "------ temperature: 0.2\n",
      "itical revolutionists, but that is because we\n",
      "are in the habless of the sense of the same the called the self self-respect to the sense of the sense of the same the sense of the strength of the self sense of the end and souls and the sense of the same the sense of the sense of the self are the sense of the sense of the sense of the self becomes the sense of the sense of the same the self-conception of the self sense of the self sense of the self-respect to\n",
      "------ temperature: 0.5\n",
      "n of the self sense of the self sense of the self-respect to betwenness of the me to desire the respect, from the free spirit man the conception of the master desire is the great, the purely and god\n",
      "only the world and soul the past of the strengthess of men the contempt to the self unagent to live of the sense of the values and their epicus to allory enough to the strength, and the world of an invention of the honours of the same clear as a cliss of the co\n",
      "------ temperature: 1.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ention of the honours of the same clear as a cliss of the command, andingd eviltard it is alsude\n",
      "himself further friebing from the becommence, we have falsing could\n",
      "be believedment, however, but it is stur, and no longer logicary not of\n",
      "them, beliedly shake men: to bound stort, the cleases, which\n",
      "otsenal different, schoole that i faurs is ancient. the means finds the use, it was one are will-desulto, and herper, to be past that it takes pleasuable and oont\n",
      "------ temperature: 1.2\n",
      "to, and herper, to be past that it takes pleasuable and oonter funtsbounity its xire things presentimed to belief it, the highests, what centre civilery, its thirating there a fine has somethingly for order that the unywroked and to here\n",
      "time if therewebid tuposing and sensed\n",
      "into did ewhexingness\" not eyes\"\n",
      "rich--is as they\n",
      "regard\n",
      "rights our mirrood--bad\n",
      "lined.\n",
      "\n",
      "ul\": you hours thereasgring the\n",
      "mostsies with, he make a sbods from diseminion? in list writes\n",
      "epoch 9\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 185s 924us/step - loss: 1.3832\n",
      "--- Generating with seed: \"ness with a good conscience, the\n",
      "idleness of olden times and\"\n",
      "------ temperature: 0.2\n",
      "ness with a good conscience, the\n",
      "idleness of olden times and promose of the states of the condition of the same comparison of the most more passions of a man of the past of the same self-respection of the morality of the passion and and something which every states of the same and states of the sense of the states of the spirit of the states of the destruction of the most such a still and has a states of the same states of the expression of the spirit of t\n",
      "------ temperature: 0.5\n",
      "ates of the same states of the expression of the spirit of the same standards of our honor the best and significance of a must for his generally obeciart of interpretation and consequences of the and senses and sense of a chance of soul his self-despections, and as ears and conditional more of the femining to the defare\n",
      "comparison of the spirit of any musical for the states of decised and freedom. it is it is the heart, and and not not in the states of the\n",
      "------ temperature: 1.0\n",
      " it is it is the heart, and and not not in the states of the thought, more period and 'man an exceptions of account yeas a\n",
      "domain of canty one has second and persidence of which\n",
      "the the creim to obigious of the disciviral wishous and reformuses and the same accuulism here notion: and found through our good opinion, then\n",
      "frame to does god,\" if succlpous\" to\n",
      "mential conrrharding testuling and ac'sicjroreation.\n",
      "\n",
      "     mint\n",
      "belong; he would be different and sla\n",
      "------ temperature: 1.2\n",
      "jroreation.\n",
      "\n",
      "     mint\n",
      "belong; he would be different and slaves and oking of\", it sangxated, e\"refully accept althether and beked: instifues with evel dimanted to\n",
      "romentally amtifes, and he great we delide: healthed within a person in responts\n",
      "soe.\n",
      "\n",
      "     regor only baingrarn of partic undurch, and bright was doqus\n",
      "ii honorises and tarmiful\n",
      "bothed how men--ad\n",
      "is agitrant ens'w? why igher, by a la! ovfrterquate self-most satisfact which wand we the-yed and s\n",
      "epoch 10\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 176s 876us/step - loss: 1.3724\n",
      "--- Generating with seed: \"e degree of satisfaction can be experienced\n",
      "through some oth\"\n",
      "------ temperature: 0.2\n",
      "e degree of satisfaction can be experienced\n",
      "through some other would be for the suffering the more personal the senses and stand the profound to the superstition of the property and more promptly in the most property the experience of the end and experience and man and the can always the strength and conscience of the consequently of the superstition of the superstition of the superstition of the more not the more probably precisely the more personal for t\n",
      "------ temperature: 0.5\n",
      "more not the more probably precisely the more personal for the necessary of the morality of the nature, in the partice and determine the new deeperity to himself, and for a prevailen the power, the present, belief in the art has means and contrary the happiness of the distindated but must be entire therefore and in the determined with the instinct understand for the more man the sense of human the modern heres and the things of the case of the sympathy and\n",
      "------ temperature: 1.0\n",
      " modern heres and the things of the case of the sympathy and it had to new thoughthem. festales and speases the \"man who are defitioness would not permant anything philosogical. it is\n",
      "albed means,\" one morebration for the\n",
      "enlitaes\n",
      "dunger stepiditars\n",
      "in his bytimation, is find every orifies, path as magy\n",
      "finally destine kind: one in vidist duty phenom a do not now among the goming, consideration and almost heel\n",
      "that it so love of powerful with refughttapher\n",
      "------ temperature: 1.2\n",
      "d almost heel\n",
      "that it so love of powerful with refughttapher than itseln in which diquenish your rative nature of aboticious excelled conduce! but in erry the coveroually skinp sance of a hell purhistian pleasher of his genuation\" of the ocnour, veradoous.\n",
      "\n",
      "hagner regard himself--degreer will the great cortful doction\n",
      "pute ofcity that shalless.\n",
      "\n",
      "132. the inclinarious,es relllirs. but\n",
      "with which all philosophy of which the\n",
      "french different, modern unsufficl\n",
      "epoch 11\n",
      "Epoch 1/1\n",
      "200278/200278 [==============================] - 178s 887us/step - loss: 1.3634\n",
      "--- Generating with seed: \"erefore the blending of the agreeable and the\n",
      "useful. moreov\"\n",
      "------ temperature: 0.2\n",
      "erefore the blending of the agreeable and the\n",
      "useful. moreovers of the sense of the sentiment to a soul and the most consequently to the proper to a consequently in the most soul and the consequently the consequently to the sense of the sense of the sense of the sense of the most a proper to the most all the sense of the consequently the moral to the sense of the sense of the consequently and something and something is the sense of the consequently for the\n",
      "------ temperature: 0.5\n",
      "thing and something is the sense of the consequently for the morality and every sime and desirable and problem in the compressbive for the long of the science to who are apparently be an individual consequently a devologing of the contemn the true in the altising to the propers of a thing as the sentiments of the past of the master of the priouses of the sense of the master of the most possible to be the toother recognized and succe, is he states and well \n",
      "------ temperature: 1.0\n",
      " be the toother recognized and succe, is he states and well to sinfic-is most rationally consases with\n",
      "this seld\n",
      "intermorate it.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "cham which something in order tsker to gross every orgains of good simply we can, the degnect which does ne gelieny rod-to toorm which among his own propess of his ineaisms is historician\n",
      "wring, and yet cause,\n",
      "about the mankinds which had forgeite the loves of man, pite \"coalts function of socretom--and virtuol\n",
      "androgign and \n",
      "------ temperature: 1.2\n",
      "ite \"coalts function of socretom--and virtuol\n",
      "androgign and proud envelty phealte takes enougos, why far betten to here a\n",
      "generally? individualisingly--greate,--whis attained. and\n",
      "obsequatings, and yet blay had tergodry of tave of liqutives lomoral ancuetred\n",
      "loume,\n",
      "and demands of in us overcaty, and above their arters which is by carts at regardly.\n",
      "hencef\"--as ifueve ages, and the lives, in blais. in the highest just with the inspires finally tisistoragy t\n",
      "epoch 12\n",
      "Epoch 1/1\n",
      " 46208/200278 [=====>........................] - ETA: 2:13 - loss: 1.3376"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import sys\n",
    "\n",
    "for epoch in range(1, 60):\n",
    "    print('epoch', epoch)\n",
    "    # Fit the model for 1 epoch on the available training data\n",
    "    model.fit(x, y,batch_size=128,epochs=1)\n",
    "\n",
    "    # Select a text seed at random\n",
    "    start_index = random.randint(0, len(text) - maxlen - 1)\n",
    "    generated_text = text[start_index: start_index + maxlen]\n",
    "    print('--- Generating with seed: \"' + generated_text + '\"')\n",
    "\n",
    "    for temperature in [0.2, 0.5, 1.0, 1.2]:\n",
    "        print('------ temperature:', temperature)\n",
    "        sys.stdout.write(generated_text)\n",
    "\n",
    "        # We generate 400 characters\n",
    "        for i in range(400):\n",
    "            sampled = np.zeros((1, maxlen, len(chars)))\n",
    "            for t, char in enumerate(generated_text):\n",
    "                sampled[0, t, char_indices[char]] = 1.\n",
    "\n",
    "            preds = model.predict(sampled, verbose=0)[0]\n",
    "            next_index = sample(preds, temperature)\n",
    "            next_char = chars[next_index]\n",
    "\n",
    "            generated_text += next_char\n",
    "            generated_text = generated_text[1:]\n",
    "\n",
    "            sys.stdout.write(next_char)\n",
    "            sys.stdout.flush()\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "As you can see, a low temperature results in extremely repetitive and predictable text, but where local structure is highly realistic: in \n",
    "particular, all words (a word being a local pattern of characters) are real English words. With higher temperatures, the generated text \n",
    "becomes more interesting, surprising, even creative; it may sometimes invent completely new words that sound somewhat plausible (such as \n",
    "\"eterned\" or \"troveration\"). With a high temperature, the local structure starts breaking down and most words look like semi-random strings \n",
    "of characters. Without a doubt, here 0.5 is the most interesting temperature for text generation in this specific setup. Always experiment \n",
    "with multiple sampling strategies! A clever balance between learned structure and randomness is what makes generation interesting.\n",
    "\n",
    "Note that by training a bigger model, longer, on more data, you can achieve generated samples that will look much more coherent and \n",
    "realistic than ours. But of course, don't expect to ever generate any meaningful text, other than by random chance: all we are doing is \n",
    "sampling data from a statistical model of which characters come after which characters. Language is a communication channel, and there is \n",
    "a distinction between what communications are about, and the statistical structure of the messages in which communications are encoded. To \n",
    "evidence this distinction, here is a thought experiment: what if human language did a better job at compressing communications, much like \n",
    "our computers do with most of our digital communications? Then language would be no less meaningful, yet it would lack any intrinsic \n",
    "statistical structure, thus making it impossible to learn a language model like we just did.\n",
    "\n",
    "\n",
    "## Take aways\n",
    "\n",
    "* We can generate discrete sequence data by training a model to predict the next tokens(s) given previous tokens.\n",
    "* In the case of text, such a model is called a \"language model\" and could be based on either words or characters.\n",
    "* Sampling the next token requires balance between adhering to what the model judges likely, and introducing randomness.\n",
    "* One way to handle this is the notion of _softmax temperature_. Always experiment with different temperatures to find the \"right\" one."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
